Photographing Device and Photographing Method

ABSTRACT

A photographing device for photographing a monitored area has a state diagnosis of a camera using a stereo camera configured by two or more cameras. A diagnostic pattern database is provided in which information on a plurality of images having different photographing conditions obtained by the stereo camera are stored in advance as diagnostic patterns. Disparity information is acquired based on image data from the stereo camera and a state diagnosis of the camera is performed by determining whether the camera is abnormal or normal based on the disparity information with reference to the information on the plurality of images having different photographing conditions stored in the diagnostic pattern database. Upon receiving a result of the state diagnosis, at least one of illumination control, shutter control, and posture control of the camera or the stereo camera is performed.

TECHNICAL FIELD

The present invention relates to a photographing device and aphotographing method used for monitoring.

BACKGROUND ART

In related fields, a video monitoring system which has a function ofrecognizing an object such as a person or a vehicle appearing in amonitored area or measured by performing image processing on a videoacquired from a camera is proposed.

By using a detection result of the camera, the video monitoring systemcan, for example, display a warning or an icon on some display device,or sound a buzzer or the like to call attention of monitoring staff.Therefore, the video monitoring system contributes to reduce burden in arelated-art monitoring task in which a confirming operation is alwaysrequired.

In addition, it is also common to provide a camera system on a movingobject such as a car or a robot and use the camera system in safedriving control for recognizing surroundings, or in a service robot forsupporting a person.

Further, in order to improve accuracy or the like, a video monitoringsystem is proposed in which a stereo camera is used to measure adistance to a subject so that a more stable recognition function isachieved compared to a case where only an image is used.

In addition, a stereo camera which is not influenced by severeconditions such as backlight, darkness, or external environments such asroad surface conditions is proposed in an in-vehicle imaging device orthe like (refer to PTL 1). In a technique described in PTL 1, disparityimage data is generated based on stereo image data from the stereocamera, and whether the camera is in a normal state or an abnormal stateis determined by determining a normal or abnormal state of the disparityimage data.

PRIOR ART LITERATURE Patent Literature

PTL 1: JP-A-2014-6243

SUMMARY OF INVENTION Technical Problem

However, in a related-art video monitoring system, desired detectionaccuracy cannot be ensured when, for example, some abnormality occurs ina camera, or dirt, attachment or the like is adhered to the camera.Therefore, reliability of the video monitoring system is impaired due toan abnormality of the camera or the like.

In addition, an intruder who performs an illegal action such as theftmay change a photographing angle of the camera, or may place a blockingobject in front of the camera, or may damage a camera lens so as not toleave video evidence, thereby causing a problem.

Further, in the technique described in PTL 1, since the normal andabnormal states of the camera are classified based on distribution ofthe disparity image data, the classification works only when disparityis available. For this reason, in a case where disparity image datacannot be obtained in a dark place where photographing is difficult, orin a case where luminance distribution at the time of photographing isuniform in a plane and no change can be observed on an image, it isdifficult to clearly classify the normal and abnormal states of thecamera.

Further, in a case where such a camera device is used in a securityfield, even if the camera device is normal, the video monitoring systemmay not operate normally when an intruder intentionally changes anorientation of the camera or covers the lens of the camera. Therefore,it is impossible to prevent a crime of an intruder or the like inadvance simply by classifying the normal or abnormal state of the cameradevice.

Further, in the case described in PTL 1, the abnormal state isdetermined in a case where disparity information cannot be normallyobtained, and in a case where disparity information is normallyobtained, even if dirt adheres to the lens, the dirt will be neglected,and the dirt will not be determined as dirt. That is, if the lens of thecamera is originally adhered with dirt, the lens may be determined asnormal although the lens should be determined as abnormal, therebycausing an error. Therefore, the state of the camera is not determinednormally.

An object of the invention is to improve the reliability of an entiresystem including a camera device without reducing a security level whenrecognizing an object by image processing.

Solution to Problem

In order to solve the above problems, for example, a configurationdescribed in the claims is adopted.

In order to solve the above problems, a photographing device of theinvention includes: a stereo camera which is configured by two or morecameras; an image processing unit which acquires disparity informationbased on image data from the stereo camera; and a diagnostic patterndatabase which stores information on a plurality of images havingdifferent photographing conditions obtained by the stereo camera asdiagnostic patterns in advance.

The photographing device of the invention further includes: a statediagnosis unit which determines a state of a camera and performs statediagnosis based on the disparity information obtained from the imageprocessing unit with reference to the information on the plurality ofimages having different photographing conditions stored in thediagnostic pattern database; and a camera control unit which performs atleast one of illumination control, shutter control, and posture controlof the stereo camera upon receiving a diagnosis result of the statediagnosis unit.

A photographing method of the invention includes: a step ofphotographing a monitored area by a stereo camera to obtain data of leftand right images; a step of acquiring, by an image processing unit,disparity information of a photographed image based on the data of leftand right images photographed by the stereo camera; a step of storinginformation on a plurality of images having different photographingconditions photographed by the stereo camera in a diagnostic patterndatabase in advance as diagnostic patterns; and a step of determining astate of a camera and performing camera state diagnosis by a statediagnosis unit based on the disparity information acquired by the imageprocessing unit and the information on the plurality of images havingdifferent photographing conditions stored in the diagnostic patterndatabase in advance.

Problems, configurations and effects other than the above will beapparent with reference to description of following embodiments.

Advantageous Effect

According to the invention, it is possible to stably determine whether astate of a camera is normal or abnormal without reducing a securitylevel. In addition, by appropriately switching the state of the camerato an appropriate mode, even when performance of the camera is reduceddue to some factors, application can be stably executed. Further, byexecuting processing corresponding to the state of the camera,reliability of an entire system can be improved.

Problems, configurations and effects other than the above will beapparent with reference to description of following embodiments.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a system block diagram showing an entire configuration of aphotographing device according to an embodiment of the invention.

FIG. 2 is a block diagram for explaining an internal function of astereo camera used in the photographing device according to theembodiment of the invention.

FIG. 3 is an external view of the stereo camera used in thephotographing device according to the embodiment of the invention.

FIG. 4 is a function block diagram showing a function of an imageprocessing unit used in the photographing device according to theembodiment of the invention.

FIG. 5 is a function block diagram showing a function of a statediagnosis unit used in the photographing device according to theembodiment of the invention.

FIG. 6 is a diagram showing an example of an environment measured by thephotographing device according to the embodiment of the invention.

FIG. 7 is a diagram showing a control pattern of the stereo camera inthe photographing device according to the embodiment of the invention.

FIG. 8 is a diagram showing an example of a detection result obtained inthe photographing device according to the embodiment of the invention.

FIG. 9 is a diagram showing a state transition of a processing mode inthe photographing device according to the embodiment of the invention.

FIG. 10 is a diagram showing a diagnostic pattern and a determinationresult for a state diagnosis in the photographing device according tothe embodiment of the invention.

DESCRIPTION OF EMBODIMENTS Entire Configuration of Embodiment of theInvention

Hereinafter, an example of a photographing device (hereinafter referredto as “the example”) according to an embodiment of the invention will bedescribed with reference to the drawings.

The invention is a photographing device generally applicable to a camerasystem for monitoring, and a stereo camera device will be describedbelow as an example.

FIG. 1 is a block diagram showing a configuration of an entire systemusing a stereo camera device as an example of the photographing deviceof the example.

The photographing device of the example includes: a stereo camera 10; animage processing unit 20; a state diagnosis unit 30; a storage device 40(hereinafter, referred to as a “diagnostic pattern database 40”)including a database in which a diagnostic pattern is stored; a cameracontrol unit 50; and an output unit 60.

The diagnostic pattern database 40 stores a plurality of diagnosticpatterns photographed in advance, and the state diagnosis unit 30determines whether a camera is normal or abnormal by referring to thediagnostic patterns using disparity information obtained from the stereocamera 10 as a trigger. Details thereof will be described later.

[Configuration of Stereo Camera 10]

FIG. 2 shows a specific configuration of the stereo camera 10 used inthe example. As shown in FIG. 2, the stereo camera 10 includes: a lensunit 201, an imaging unit 202, and an image generating unit 203. Thestereo camera 10 further includes: an imaging control unit 204, anadjustment unit 205, a light projecting unit 206, an illuminationcontrol unit 207, and a posture control unit 208. Although the stereocamera 10 requires two (a pair of) lens units 201 and two photographingunits 202 in order to obtain left and right images, only one lens unit201 and one imaging unit 202 are shown in FIG. 2 for simplification.

The lens unit 201 is an optical lens for photographing. The imaging unit202 is a so-called image sensor, and an image sensor including aphotographing element (not shown) such as a complementary metal oxidesemiconductor (CMOS) or a charge coupled device (CCD) can be used.

The imaging unit 202 converts a video took from the lens unit 201 intoan electrical signal, and supplies the electrical signal to the imagegenerating unit 203. The image generating unit 203 performspredetermined processing such as noise reduction, color correction, andimage synthesis on an image sent from the imaging unit 202, and suppliesprocessed image data to the subsequent image processing unit 20.

The imaging control unit 204 controls parameters such as a shutter, anaperture, and a zoom of a camera in the imaging unit 202 based on acontrol signal from the adjustment unit 205 to be described later.

The light projecting unit 206 is a member which emits light to a subjectto be photographed for illumination. Usually visible light is used, butother than visible light, near infrared light or far infrared light mayalso be used depending on the subject. The light projecting unit 206 isparticularly effective in photographing an object in a dark area, andthus even an image of low luminance can be stably acquired.

The illumination control unit 207 controls an intensity level of lightemitted from the light projecting unit 206 based on a control signalfrom the adjustment unit 205 to be described later. In addition,depending on a result of a state diagnosis in the state diagnosis unit30 in FIG. 1, for example, control such as switching between left andright illumination may be performed.

The posture control unit 208 performs control to change a posture, anorientation or the like of the camera. Accordingly, the stereo camera 10of the example can be configured as a so-called pan-tilt camera which iscapable of dynamically changing the orientation of the camera. That is,photographing is performed again after the posture and the orientationof the camera is changed by the posture control unit 208, and therefore,an image different from the previous image can be obtained. The pan-tiltcamera refers to a camera capable of dynamically changing both of anangle in a horizontal direction (panning angle) and an angle in avertical direction (tilting angle), and may also be a camera capable ofchanging an angle in only the horizontal direction or the verticaldirection.

The adjustment unit 205 controls functions of each of the imagegenerating unit 203, the imaging control unit 204, the illuminationcontrol unit 207, and the posture control unit 208 based on a controlsignal from the camera control unit 50 shown in FIG. 1. That is, as aresult of the diagnosis of the state of the camera in the statediagnosis unit 30 in FIG. 1, a control signal is supplied to the cameracontrol unit 50 from the state diagnosis unit 30. Then, the cameracontrol unit 50 sends various control signals to each part of the stereocamera 10 via the adjustment unit 205, and performs the above-describedcamera control.

FIG. 3 shows a configuration of the stereo camera 10 shown in FIG. 1 andFIG. 2 as an integrated device. Since two lens units 201 are required inFIG. 2, the lens units are shown as a lens 301 and a lens 302 in FIG. 3.In addition, the light projecting unit 206 of FIG. 2 is shown as a lightprojecting unit 303 in FIG. 3. A light emission diode (LED), forexample, is used as the light projecting unit 303, and light (emittedlight) projected from the light projection unit 303 is reflected by anobject serving as a subject, and is incident on the lens units 301 and302 of the stereo camera 10. Accordingly, left and right images of thesubject are obtained from the stereo camera 10.

As shown in FIG. 3, in consideration of an amount of light to beemitted, the light projecting unit 303 is provided with a plurality ofLEDs. By controlling the plurality of LEDs in accordance with anorientation of light emitted from the light projecting unit 303 or adistance to the subject, the light having the adjusted light amount fromthe light projecting unit 303 is applied toward a specific area.

A stage 304 of the stereo camera 10 in FIG. 3 is a part where the cameracontrol unit 50 of FIG. 2 is disposed, and the stage 304 has amanipulation function capable of controlling a posture of the camerawith two axes in a horizontal direction and a vertical direction, forexample. Therefore, the camera control unit 50 disposed on the stage 304can direct the orientation of the camera in a specific direction. Inaddition, as will be described later, when the camera is determined tobe abnormal, the direction of the camera is freely changed so as toperform photographing again.

Referring back to FIG. 1, the entire configuration of the photographingdevice of the example will be continuously described. Data of left andright images acquired by the stereo camera 10 is transmitted to theimage processing unit 20. In the image processing unit 20, a process isexecuted as shown in FIG. 4 which will be described later, and aprocessing result is transmitted to the state diagnosis unit 30.

As shown in FIG. 1, the photographing device of the example includes thediagnostic pattern database 40. The diagnostic pattern database 40 isstored with diagnostic patterns for classifying a state of the camera inadvance based on information on a plurality of images having differentphotographing conditions photographed by the stereo camera 10.

In the photographing method of the example, this is a step of storingthe information on the plurality of images having differentphotographing conditions photographed by a stereo camera in a diagnosticpattern database as diagnostic patterns in advance.

As will be described later with reference to FIG. 5, the state diagnosisunit 30 reads the diagnostic patterns stored in the diagnostic patterndatabase 40 in advance, and performs diagnosis processing on output datafrom the image processing unit 20 with reference to the diagnosticpatterns. A processing result of the state diagnosis unit 30 is sent tothe output unit 60, and is usually presented as a video using a terminaldevice such as a liquid crystal display device or a cathode ray tube(CRT) display device.

The diagnosis result output to the output unit 60 may be displayed as aRed-Green-Blue (RGB) monitor output, or may be transmitted to a centralserver via a network as a data file output. Further, it is also possibleto call attention to the surroundings by using an alarm device such as abuzzer.

In addition, the state diagnosis unit 30 sends the diagnosis result tothe camera control unit 50 as control data for the stereo camera 10. Thecamera control unit 50 controls the stereo camera 10 based on thecontrol data sent from the state diagnosis unit 30. That is, asdescribed above, the imaging control unit 204, the image generating unit203, the illumination control unit 207, and the posture control unit 208are controlled according the control signal from the camera control unit50 through the adjustment unit 205 of the stereo camera 10.

[Configuration and Function of Image Processing Unit 20]

FIG. 4 is a function block diagram illustrating functions of the imageprocessing unit 20 in FIG. 1 as blocks. As shown in FIG. 4, the imageprocessing unit 20 includes a camera image acquiring unit 401, adistortion correcting unit 402, a disparity measuring unit 403, and anobject detecting unit 404.

In the photographing device of the example, positional relationshipsbetween the camera and the real environment, such as a distance betweenthe left and right lenses in the stereo camera 10, focal length of alens, height of the camera, and an orientation of the camera areacquired in advance as advance preparation. That is, it is necessary toobtain in advance, by calibration, a coordinate system indicating aposition of the camera in the real environment in which the camera isprovided.

External parameters such as the orientation and the posture of thestereo camera 10 can be obtained in advance by projecting a pattern suchas a checker chart onto the real environment or by attaching the patternto the lens of the camera. This method is described in, for example, Z.Zhang, “A flexible new technique for camera calibration”, IEEETransactions on Pattern Analysis and Machine Intelligence,22(11):1330-1334, 2000. Accordingly, the positional relationship betweenthe stereo camera 10 and the object to be photographed is known, and theposition of the detected object in the real environment can bespecified.

Similarly to the external parameters, internal parameters such asdistortion of the lens of the stereo camera 10 and a focal length canalso be obtained in advance by projecting or attaching a pattern such asa checker chart.

Although the stereo camera 10 is usually used as a passive sensor, apattern irradiation type active sensor capable of acquiringthree-dimensional information may be used instead of such stereo camera10. It is also possible to use an active sensor, which is known as aTime-of-Flight type active sensor, capable of measuring a time requiredfor emitted light such as a laser beam, to be reflected from a subjectand returns back. In this way, by using the pattern irradiation typesensor or the Time-of-Flight type active sensor, the position of theobject can be easily specified as compared with the passive sensor.

As shown in FIG. 4, the image processing unit 20 acquires the left andright camera images from the stereo camera 10 in the camera imageacquiring unit 401. Next, the distortion correcting unit 402 correctsdistortion of the left and right camera images acquired by the cameraimage acquiring unit 401 based on predetermined parameters. Thedistortion correction is executed based on the internal and externalparameters of the camera acquired in advance by a rectificationprocessing in a parallel stereo processing.

In general, the “parallel stereo processing” is performed on the leftand right camera images of the stereo camera 10 at the same time on theleft and right, but distortion may occur in one or both of the lensesdepending on characteristics of the lenses or the like, and a grid thatshould originally be parallel straight lines may be distorted and becomecurved lines having a curvature. Therefore, it is necessary to comparethe left and right images after the distortion correction which changesthe curved lines having the distortion into parallel straight lines.Such processing is the “rectification processing”, which facilitatescollation of the right and left camera images and makes it easier toobtain disparity image data.

The stereo camera images subjected to the distortion correction at thedistortion correcting unit 402 are input to the disparity measuring unit403 so as to be subjected to disparity measurement. The disparitymeasuring unit 403 obtains left and right disparity information(deviation amount of left and right positions) for each pixel of theleft and right camera images. In addition, a distance to the subject canbe calculated based on a principle of triangulation from the disparityinformation obtained for each pixel. In this sense, since the disparityinformation and the distance information are equivalent, hereinafter,the distance information will be collectively referred to as the“disparity information”.

In the photographing method of the example, this is a step ofphotographing a monitored area to obtain left and right disparity data.

Output of the disparity measuring unit 403 is sent to the objectdetecting unit 404, where a position of a person such as an intruder oran object in the vicinity of the camera is detected, for example.Specifically, the object detecting unit 404 can easily detect an objectusing methods such as obtaining a difference between the image data fromthe disparity measuring unit 403 and background information photographedin advance.

A detection result of the object detecting unit 404 is sent to the statediagnosis unit 30 shown in FIG. 1. As described above, the processingfrom the camera image acquiring unit 401 to the object detecting unit404 of the image processing unit 20 is a step of acquiring the disparityinformation based on the data of the left and right images photographedby the stereo camera.

In the above processing, a person or an object such as a vehicle isdetected, but if the camera side is adhered with dirt, consequently, theobject to be detected may not be detected normally, and the dirt or thelike which should not be detected may be erroneously detected as aperson. Alternatively, a disturber may place a blocking object or thelike in front of the lens of the camera, and in such a case, the objectto be detected may not be detected in a similar way. In such cases, itis necessary to take measures to clearly distinguish whether an objectphotographed by the camera is an actual intruder or dirt such asrubbish.

[Configuration and Function of State Diagnosis Unit 30]

Next, the configuration of the state diagnosis unit 30 and functionsthereof will be described with reference to FIG. 5.

As shown in FIG. 5, the state diagnosis unit 30 includes a resultacquiring unit 501, a data acquiring unit 502, and a state determiningunit 503.

The result acquiring unit 501 acquires the disparity information whichis a processing result from the image processing unit 20.

In addition, the data acquiring unit 502 acquires a diagnostic patternwhich is prepared in advance based on predetermined conditions andstored in the diagnostic pattern database 40. Then, the acquireddiagnostic pattern is sent to the state determining unit 503 togetherwith the disparity information input from the result acquiring unit 501.

Further, the state determining unit 503 determines the state of thecamera based on the disparity information from the result acquiring unit501 with reference to the diagnostic pattern acquired by the dataacquiring unit 502. A determination result in the state determinationunit 503 is output to the camera control unit 50 and the output unit 60shown in FIG. 1.

The processing of the result acquiring unit 501, the data acquiring unit502, and the state determining unit 503 of the state diagnosis unit 30is a step of determining the state of the camera and performing statediagnosis for the camera.

Next, the processing of the state diagnosis unit 30 shown in FIG. 5 willbe described in detail with reference to FIG. 6 to FIG. 8.

FIG. 6 is a schematic diagram showing an example of three typicalobjects photographed by the stereo camera 10.

That is, in FIG. 6, three objects including a small object A in thevicinity of the camera, a distant large object B, and a distant lightemitter C are provided with respect to the stereo camera 10. It isassumed that the object A is at a location close to the stereo camera 10(1 m in the vicinity), and the object B is at a location (10 m away) farfrom the stereo camera 10. In addition, it is assumed that the lightemitter C is an object that emits light by itself (self-emitting object)at a location farther than the object B.

As shown in FIG. 6, for the small object A placed at a distance of 1 mfrom the stereo camera 10 and the large object B placed at a distance of10 m from the stereo camera 10, sizes of images photographed by thestereo camera 10 are substantially the same. In addition, since thelight emitter C of FIG. 6 emits light to the stereo camera 10 by itself,the light emitter C may be erroneously recognized as an object such asan intruder. This example will be described below as a case where aperson such as an intruder or an object such as a dangerous object isphotographed in a situation where photographing is difficult withoutillumination especially in a dark place at night.

[Photographing Pattern Classification and Operation Sequence]

FIG. 7 shows an operation sequence of the imaging control unit 204 andthe illumination control unit 207 of the stereo camera 10. In FIG. 7,the horizontal axis indicates a time axis, and each section of thesections (1) to (5) indicates light projection and shutter controlpatterns (1) to (5) (hereinafter referred to as “photographing patterns(1) to (5)”) in the photographing device of the example.

Here, for illumination control, light projected from the lightprojecting unit 206 in FIG. 2 is simplified and described as two typesincluding a “low intensity” illumination for photographing a distance of1 m in the vicinity and a “high intensity” illumination forphotographing a distance of 10 m away. In addition, the shutter controlis simplified into two types including a case when an exposure time ofthe shutter is long and a case when the exposure time is short.

The photographing pattern (1) is a pattern in which the light projectingunit 206 of the stereo camera 10 photographs an area near the stereocamera 10 (1 m away in the vicinity) with weak illumination and a longexposure time.

The photographing pattern (2) is a pattern in which photographing isperformed with no light projected from the light projection unit 206 ofthe stereo camera 10 and with a long exposure time.

The photographing pattern (3) is a pattern in which the light projectingunit 206 of the stereo camera 10 photographs an area far from the stereocamera 10 (10 m away in the distance) with strong illumination and along exposure time.

The photographing pattern (4) is a pattern in which photographing isperformed with no light projected from the light projection unit 206 ofthe stereo camera 10 and with a short exposure time.

The photographing pattern (5) is a pattern in which the light projectingunit 206 of the stereo camera 10 photographs a location 1 m away fromthe stereo camera 10 with weak illumination and a short exposure time.

The “image processing” shown in FIG. 7 indicates that the timing of theimage processing in the image processing unit 20 of FIG. 4 is executedwith a predetermined processing cycle. This image processing shows thatin consideration of the time for generating image data, this timingbecomes the next cycle of shutter operation, that is, the processing ofthe image photographed by the photographing pattern (1) is processedduring a period when the next photographing pattern (2) is beingexecuted.

FIG. 8 shows an image obtained by photographing the object A, the objectB, and the light emitter C of FIG. 6 using the respective photographingpatterns (1) to (5) of FIG. 7.

In the photographing pattern (1), since the object A is close to thecamera and the exposure time is long, a clear image is detected even ifthe light projected from the light projection unit 206 is weakillumination. In FIG. 8, the clear image is represented by a blackellipse. Hereinafter, such detection is defined as “clear detection”.

In addition, in the photographing pattern (1), since the object B is ata position 10 m away from the camera, the object B is not detected withthe light projected from the light projection unit 206 being weakillumination. Further, since the light emitter C is an object that emitslight by itself, even if the light projected from the light projectionunit 206 is weak, the light emitter C is detected as a faint blurredlight. In this way, a detection of an object with faint blurred light isdefined as “faint detection” here. In FIG. 8, the “faint detection” isrepresented by a gray ellipse whose color is an intermediate color.

The photographing pattern (2) is a case in which no light is projectedfrom the light projecting unit 206 and the exposure time is long. Evenin a case where no light is projected from the light projecting unit206, the object A at a position close to the camera is slightly detectedbecause the exposure time is long. Since in a detection result of thisdetection, detected portions and portions that are not detected aremixed as noise is included, hereinafter, such detection is defined as“slight detection”. In FIG. 8, the “slight detection” is represented byan ellipse having dotted outline.

Since the object B is far away (10 m ahead) from the camera, the objectB is not detected in a state where no light is projected. In addition,the light emitter C is detected as a clearer image as compared with thephotographing pattern (1) in which light is projected from the lightprojecting unit 206 since there is no light projection. This is becausethe light emitter C is an object that emits light by itself, andtherefore, the clearer image is detected without illumination light.

The photographing pattern (3) is a case in which illumination light withhigh intensity is emitted from the light projecting unit 206 and theshutter exposure time is long. In this case, the detection of the nearbyobject A and the detection of the distant object B are both “cleardetection”. This is because the light projection for illumination isstrong enough to reach a distant area. However, in the case of the lightemitter C, since the emitted light is canceled by the illuminationlight, the light emitter C is hardly detected or detected as a blurredthin image. That is, an image similar to the image of the so-called“slight detection”.

The photographing pattern (4) is a case where no light is projected fromthe light projecting unit 206 and the shutter exposure time is short. Inthe photographing pattern (4), since no light is projected and theexposure time is short, neither the nearby object A nor the distantobject B is detected. However, for the light emitter C, the self-emittedlight is clearly detected as no light is projected for illumination.

Finally, the photographing pattern (5) is a pattern in which the lightemission of the light projecting unit 206 is weak and the shutterexposure time is short. Therefore, the object A placed in the vicinityis detected as “faint detection”. However, the object B placed far away,such as 10 m away, is not detected. In addition, the self-emitted lightof the light emitter C is detected as “faint detection” because no lightis projected by the light projecting unit 206.

Next, the photographing patterns (1) to (5) described above are comparedwith each other and the photographing states in each of thephotographing patterns (1) to (5) will be described focusing on each ofthe nearby object A, the distant object B, and the light emitter C.

The object A in the vicinity of the camera is clearly detected either inthe weak illumination of the photographing pattern (1) or in the strongillumination of the photographing pattern (3), because the exposure timeis long. However, since no light is projected in the photographingpatterns (2) and (4), the object A is detected as “slight detection” inthe photographing pattern (2) having a long exposure time, and is notdetected in the photographing pattern (4) having a short exposure time.In the photographing pattern (5), since the exposure time is short, theobject A is detected as “faint detection” in proportion to the shortexposure time, as compared with the “clear detection” obtained inphotographing pattern (1) with a long exposure time.

Next, in the cases of the object B which is distant from the camera, theobject B cannot be detected except in the photographing pattern (3) withstrong illumination and a long exposure time.

In addition, in the cases of the distant light emitter C, “faintdetection” is obtained in the photographing patterns (1) and (5) whereillumination is projected, while “slight detection” is obtained in thephotographing pattern (3) because the strong illumination makes thelight emitter C inconspicuous. Since the light emitter C isself-emitting, “clear detection” is obtained in the photographingpatterns (2) and (4) in which the light projection is turned off.

A method of detecting three kinds of objects represented by the objectA, the object B, and the light emitter C based on the photographingpatterns (1) to (5) is described above.

In the photographing device of the example, since the stereo camera 10is used, if “clear detection” of an object is obtained as shown by ablack ellipse in FIG. 8, the disparity measuring unit 403 can correctlymeasure a distance to the object based on the disparity information.

For example, in the photographing pattern (1), the distance of theobject A in the vicinity of the camera can be measured, and in the caseof the photographing pattern (3), the distances of both the object A inthe vicinity of the camera and the distant object B can be measured.

On the other hand, in the case of the light emitter C, although “cleardetection” is obtained in the photographing pattern (2) and thephotographing pattern (4), since the light emitter C is self-emitting,it is difficult to correctly recognize the shape of the light emitter,and as a result, the distance to the light emitter C may be erroneouslydetected.

In view of this, in the example, the photographing patterns (1) to (5)are selectively used, and the state determining unit 503 performs adetermination to distinguish between the light emitter C and the objectA or the object B from different image data obtained according to thephotographing patterns (1) to (5).

Therefore, even if an abnormality occurs due to disturbance such as dirtor a blocking object on the stereo camera 10, an object or a lightemitter can still be correctly determined, and it is possible tocorrectly measure what event it is.

[Processing Mode of Photographing Device]

Next, a result of the state determination in the state determining unit503 of the state diagnosis unit 30, setting of a processing mode, andstate transition thereof will be described with reference to FIG. 9 andFIG. 10.

FIG. 9 is a diagram showing the state transition of processing modes ofthe photographing device of the example, and FIG. 10 is a diagramshowing a diagnostic pattern for a state diagnosis of the photographingdevice of the example and a determination result.

Four processing modes including a standby mode S1, a diagnostic mode S2,a maintenance mode S3, and an alert mode S4 in FIG. 9 correspond to theprocessing mode 97 in FIG. 10.

The standby mode S1 in FIG. 9 corresponds to a state in which nothing isperformed. If a candidate having a certain possibility of abnormality isfound at this stage, the process switches from the standby mode S1 tothe diagnostic mode S2. After entering the diagnostic mode S2, asalready described, photographing of the object A, the object B, thelight emitter C or the like and diagnosis thereof are performed usingthe plurality of photographing patterns (1) to (5). In this case, theheight, the orientation, and the illumination of the camera are changedto acquire various camera images for diagnosis.

If an intruder, for example, is found as a result of the diagnosis inthe diagnostic mode S2, the process proceeds to the alert mode S4. Inthe alert mode S4, a display or an alarm is output to indicate that thealert mode is activated. Then, after the alarm or the like is notified,the process returns to the standby mode S1 again.

On the other hand, as a result of the diagnosis in the diagnostic modeS2, when it is determined that the candidate of the abnormality is notan intruder, but dirt on the lens, for example, the process proceeds tothe maintenance mode S3, and after some maintenance is performed, theprocess returns to the standby mode S1.

As a result of the diagnosis in the diagnostic mode S2, if only thedistant light emitter C is photographed and no special abnormality isrecognized, the process returns to the standby mode S1 again.

FIG. 10 is an example of a table showing diagnostic patterns 98 a to 98d in a case where an area is photographed based on the photographingpatterns (1) to (5) of FIG. 7. The diagnostic patterns 98 a to 98 d arestored in the diagnostic pattern database 40 shown in FIG. 1 in advance.

Reference numerals 91 to 95 in a horizontal axis of FIG. indicate imagedetermination criteria based on image processing results obtained by theimage processing unit 20 in FIG. 1 and FIG. 4. Since the determinationcriteria 91 to 95 correspond to the photographing patterns (1) to (5),in FIG. 10, the photographing patterns (1) to (5) shown in FIG. 7 andthe determination criteria 91 to 95 are associated with each other.

Each row in FIG. 10 represents a diagnostic pattern. Here, in thediagnostic patterns 98 a to 98 d, numerical values used indicate howclearly an object is detected, and “>100” (a value over 100) indicates“clear detection” in which the object is photographed very clearly andaccurately. In addition, “>20” (a value over 20) indicates “slightdetection”, and “>50” (a value over 50) indicates “faint detection”between the “clear detection” and the “slight detection”. Thesenumerical values can be freely set in accordance with conditions ofphotographing and diagnosis.

For example, in the case of the diagnostic pattern 98 a, when a controlmode of the stereo camera 10 is the photographing pattern (1), thenearby object A is clearly detected with the determination criteria 91.Then, it is determined as an “abnormal candidate” as shown by adetermination result 96. When the “abnormal candidate” is determined inthe determination result 96, the processing mode 97 switches from the“standby mode S1” to the “diagnostic mode S2” in FIG. 9.

Next, in the case of the diagnostic pattern 98 b, “clear detection” isobtained in the photographing patterns (1) and (5) in which the light isprojected, and “slight detection” is obtained in the photographingpattern (2) in which no light is projected. Nothing is detected in thephotographing pattern (4) in which light is not projected and anexposure time is short. In addition, in the photographing pattern (5) inwhich light is projected to a nearby area and the exposure time isshort, “faint detection” is obtained. In such a case, since there is ahigh possibility that attachment is adhered to the stereo camera 10, theresult “attachment” is determined in the determination result 96, andthe processing mode 97 switches to the “maintenance mode S3”.

In addition, in the case of the diagnostic pattern 98 c, “cleardetection” is only obtained in the photographing pattern (3), andnothing is detected in the other photographing patterns (1), (2), (4),and (5). In this way, when only the distant object B is detected, theresult “distant object” is determined in the determination result 96,and the processing mode 97 switches to the “alert mode S4”. This isbecause the detected object may be an intruder when something is clearlydetected in a distance of 10 m away, for example.

In the case of the diagnostic pattern 98 d, “faint detection” isobtained in the photographing patterns (1) and (5) in which the light isprojected, and “clear detection” is obtained in the photographingpatterns (2) and (4) where no light is projected. In addition, in thephotographing pattern (3) in which the light is projected to a distantarea, “slight detection” is obtained. In this case, the detection showsthe same trend as the detection result of the light emitter C of FIG. 8.Therefore, the detected object is determined as the light emitter C inthe determination result 96. Then, it is determined that the object isnot suspected of being an intruder, and the processing mode 97 remainsin the “standby mode S1”.

As described above, in the example, when detecting a state of an areamonitored by the stereo camera 10, various images are acquired byillumination control of the camera and shutter control in the cameracontrol, and the state of the monitored area is classified.

However, for example, when the light projecting unit 303 of FIG. 3 emitslight to an area in the vicinity of the camera, an object whose positionor orientation is detected as an object candidate at an initial stage isnot detected as time elapses. In such a case, the detection may bedetermined to be a false alarm. Accordingly, in a case of photographingan object in a distant area, a false alarm is less likely to bedetermined because the object is detected for a relatively long periodof time.

In other words, it is possible to determine whether the object is anearby object or a distant object by moving the orientation of thecamera. This is because, when an object is in the vicinity, movement ofa detection position increases when the camera moves along movement ofthe object, and when the object is distant, the movement of thedetection position reduces.

Although the stereo camera 10 is described in the example, it is evidentthat the same processing can be performed using two or more cameras, forexample, three cameras.

In addition, in a case where the processing mode is “maintenance”, forexample, dirt is detected on either left or right of the stereo camera10, which may cause a concern of performance degradation. In such acase, in order to ensure security, it is necessary to use a normalcamera on which dirt is not detected to continue to execute some certainprocessing without stopping the operation. At the same time, bynotifying a control center or the like, it is also possible to ensure asecurity level without stopping the system.

As an example of the photographing device of the invention, aphotographing device and a photographing method for detecting a nearbyobject, a distant object and a light emitter using a stereo camera 10based on disparity information thereof to diagnose a state of amonitored area have been described. However, the example described inthe present example is only an exemplary embodiment, and it goes withoutsaying that the invention includes other applications or modificationswithout departing from the gist of the invention as set forth in theclaims.

For example, in the above-described exemplary embodiment, photographingpatterns (1) to (5) are shown as the imaging modes, which arecombinations of the intensity of light projection and the shutterexposure time. In contrast, intensity and a direction of lightprojection, exposure time, an aperture, an orientation (posture) of acamera, and a focal length of a lens may be dynamically controlled, anda plurality of photographing patterns may be set using a plurality ofcombinations of the parameters above. Further, the acquiredphotographing pattern may be set by using not only an RGB sensor as theimage sensor, but also a sensor which is sensitive to infrared inaddition to RGB, or by changing sensor sensitivity through gainadjustment or the like.

Some or all of the above-mentioned configurations, functions, processingunits, processing means or the like may be realized by hardware, forexample, by designing in an integrated circuit. In addition, theabove-mentioned configurations, functions or the like may be realized bysoftware by interpreting and executing programs that implement eachfunction by a processor. Information of programs, tables, files or thelike for implementing each function can be placed in a recording devicesuch as a memory, hard disk, and solid state drive (SSD), or a recordingmedium such as an IC card, SD card, and DVD.

In addition, control lines and information lines show those consideredto be necessary for the description, and not all the control lines andinformation lines are necessarily shown on the product. In practice,almost all the configurations are considered to be mutually connected.

REFERENCE SIGN LIST

-   -   10: stereo camera    -   20: image processing unit    -   30: state diagnosis unit    -   40: diagnostic pattern storing unit    -   50: camera control unit    -   60: output unit    -   201, 301 and 302: lens units    -   202: imaging unit    -   203: image generating unit    -   204: imaging control unit    -   205: adjustment unit    -   206 and 303: light projecting units    -   207: illumination control unit    -   208: posture control unit    -   304: stage    -   401: camera image acquiring unit    -   402: distortion correcting unit    -   403: disparity measuring unit    -   404: object detecting unit    -   501: result acquiring unit    -   502: data acquiring unit    -   503: state determining unit

1. A photographing device comprising: a stereo camera which isconfigured by two or more cameras; an image processing unit whichacquires disparity information based on image data from the stereocamera; a diagnostic pattern database which stores information on aplurality of images having different photographing conditions obtainedby the stereo camera as diagnostic patterns in advance; a statediagnosis unit which determines a state of a camera and performs statediagnosis based on the disparity information obtained from the imageprocessing unit with reference to the information on the plurality ofimages having different photographing conditions stored in thediagnostic pattern database; and a camera control unit which performs atleast one of illumination control, shutter control, and posture controlof the stereo camera upon receiving a diagnosis result of the statediagnosis unit.
 2. The photographing device according to claim 1,wherein the state diagnosis unit diagnoses four processing modesincluding a standby mode, a diagnostic mode, an alert mode, and amaintenance mode, and switches to any one of the four processing modesin accordance with a diagnosis result.
 3. The photographing deviceaccording to claim 2, wherein the state diagnosis unit switches to astandby processing mode set in advance after switching to any one of thefour processing modes in accordance with the diagnosis result.
 4. Thephotographing device according to claim 1, wherein the camera controlunit dynamically controls intensity and direction of illumination,exposure time, aperture, camera orientation, sensor sensitivity andfocal length, and based on a combination thereof, the stereo cameraacquires the information on the plurality of images.
 5. Thephotographing device according to claim 1, wherein the image processingunit includes: a camera image acquiring unit which acquires left andright camera images from the stereo camera; a distortion correcting unitwhich corrects distortion of the camera image acquired by the cameraimage acquiring unit; a disparity measuring unit which calculates thedisparity information from the left and right camera images havingdistortion corrected by the distortion correcting unit; and an objectdetecting unit which detects an object as a subject based on thedisparity information calculated by the disparity measuring unit.
 6. Thephotographing device according to claim 5, wherein the state diagnosisunit includes: a result acquiring unit which acquires the disparityinformation input from the image processing unit; a data acquiring unit,into which the disparity information acquired by the result acquiringunit is input, and which acquires a parameter to be determined byacquiring, from the diagnostic pattern database, the information on theplurality of images having different photographing conditions obtainedby the stereo camera and stored in the diagnostic pattern database inadvance; and a state determining unit which determines a normal state oran abnormal state of the stereo camera based on the parameter to bedetermined acquired by the data acquiring unit.
 7. A photographingmethod comprising: a step of photographing a monitored area by a stereocamera to obtain data of left and right images; a step of acquiring, byan image processing unit, disparity information of a photographed imagebased on the data of left and right images photographed by the stereocamera; a step of storing information on a plurality of images havingdifferent photographing conditions photographed by the stereo camera ina diagnostic pattern database in advance as diagnostic patterns; a stepof determining a state of the camera and performing camera statediagnosis by a state diagnosis unit based on the disparity informationacquired by the image processing unit and the information on theplurality of images having different photographing conditions stored inthe diagnostic pattern database in advance.